Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique

Overview

Identifying equivalent mutants remains the largest impediment to the widespread uptake of mutation testing. Despite being researched for more than three decades, the problem remains. We propose Trivial Compiler Equivalence (TCE) that exploits the use of readily available compiler technology to address this long-standing challenge. TCE is directly applicable to real word programs and can imbue existing tools with the ability to detect equivalent mutants and a special form of useless mutants called duplicated. We present a thorough empirical study using 6 large open source programs, several orders of magnitude larger than those used in previous work, and 18 benchmark programs with hand-analysis equivalent mutants. Our results reveal that on large real word programs TCE can discard approximately one third of all the mutants. This accounts 8% of equivalent mutants and 21%* of duplicated mutants. Results with hand- analysis equivalent mutants suggest that TCE has the ability to detect as many as 30% of all the existing equivalent mutants.

Subjects

We used two sets of subjects. The first is composed of six large open source programs. In this set, we choose ‘real-world’ programs that vary in size and application domain. The second set was taken from the study of Yao et al. and it is composed of 18 subjects. The figure below shows the selected real-world programs used. The source code of the used components can be downloaded from here. subjects table

Mutation Operators

We use two sets of operators, named the ‘selective’ and the ‘empirical’ sets. The first set, proposed by Offutt et al., is composed of five operators, i.e., ABS, AOR, LCR, ROR, and UOI. We use this set due to its extensive use in literature. The second set is composed of eight operators. Three of them, i.e., AOR, LCR and ROR, are drawn from the ‘selective’ set, while, the other five, i.e., CRCR, OAAA, OBBN, OCNG and SSDL, are designed to cater the common C faults. This set was used in the studies of Andrews et al., where it was shown that it provides accurate predictions of the real fault detection ability of the test suites. A detailed description of the operators is reported in the table below.

To generate the mutants, we use MILU, an open source mutation testing tool for C. We detail exactly how the operators were applied since this is an important piece of information that differs from one tool to another. The ABS and UOI operators were only applied on numerical variables. The CRCR was applied to integer and floating numeric constants. No mutant operator was applied on the variables of the lefthand side of assignment statements; we only apply them at the right hand sides. All operators are applied recursively to all sub expressions. Further details and the implementation of the operators can be found on the github webpage of MILU.

subjects table

Downloads

Subjects

The preprocessed component for each subject and the list of functions considered in our experiments can be downloaded from the list below.

Tool

Our TCE approach has now been incorporated into the MILU mutation tool, making it the first tool that supports automated equivalent mutant detection. Milu can be downloaded from here. The mutaiton operator configuration file can be downloaded here.

To activate TCE, we need two flags, 1) the "--TCE" flag and 2) the compiler flag "-c"

Usage Example with Gzip tree.c
  • milu gzip_trees.c -m mop.txt --TCE -c "gcc -c -O2" --func-name=compress_block

The output of the example are as follow. All mutants are stored in the milu_output folder by default. The duplicated mutants and equivalent mutants will be removed in the further process of mutation testing. 

------------------------------
Generated 259 mutants
Number of compilable mutants: 245
Number of equivalent mutants: 16
Number of duplicated mutants: 65
------------------------------

Results

The results of our experiments can be downloaded here. In the results folder, mutants_0, mutants_1, mutants_2, and mutants_3 folders contain the results using gcc settings with no optimisation, -O, -O2 and -O3 respectively. There are three files in each unit result folder, 1) mutant_list.txt records types of mutants generated, 2) eq_mutant_list.txt records the detected equivalent mutants and 3) du_mutant_list.txt records the detected duplicated mutants.

Download Results